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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH{S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time nnay be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S. C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 23 October 2003 . 
2a)\Z\ This action is FINAL. 2b)M This action is non-final. 

3) n Since this application is in con(Jition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Expa/te Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) M Claim(s) 1-72 is/are pending in the application. 

4a) Of the above claim(s) 18-30 and 48-72 is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) 13 Claim(s) 1-17 and 31-47 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10) K The drawing(s) filed on 24 April 2000 is/are: a)^ accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) 0 The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, connected drawings are required In reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 .□ Certified copies of the priority documents have been received. 

2.n Certified copies of the priority documents have been received in Application No. . 

30 Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) [3 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 

Attachment(s) 

1) 13 Notice of References Cited {PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) n Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) CI Notice of Informal Patent Application (PTO-152) 

3) S Information Disclosure Statement(s) (PTO-1449) Paper No(s) 3. 6) □ Other: 
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DETAILED ACTION 
Election/Restrictions 

1. Applicant's election without traverse of Group I (claims 1-17 and 31-47) in Paper No. 8, 
filed October 23, 2003, is acknowledged. 

Information Disclosure Statement 

2. The listing of references in the specification is not a proper information disclosure 
statement. 37 CFR 1.98(b) requires a list of all patents, publications, or other information 
submitted for consideration by the Office, and MPEP § 609 A(l) states, "the hst may not be 
incorporated into the specification but must be submitted in a separate paper." Therefore, unless 
the references have been cited by the examiner on form PTO-892, they have not been 
considered. 

Claim Objections 

3. Claims 39 and 44-46 are objected to because of the following informalities: it appears 
that appHcant intended claim 39, which is similar to claim 9, to depend from claim 36, which is 
similar to claim 6. Similarly, it appears that applicant intended claims 44-46, which are similar 
to claims 14-16, to depend from claim 43, which is similar to claim 13. Appropriate correction is 
required. 

Claim Rejections - 35 USC § 112 

4. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 
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5. Claims 39 and 44-46 are rejected under 35 U.S.C. 1 12, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

6. Claim 39 recites the limitation "said singular value decomposition" in line 1-2. There is 
insufficient antecedent basis for this Umitation in the claim. 

7. Claim 44 recites the limitation "wherein in said (3) said plurality of sorted vectors" in line 
1-2 and "said first cluster" in line 4. There is insufficient antecedent basis for these limitations in 
the claim. 

8. Claim 45 recites the limitation "said first cluster" in line 1. There is insufficient 
antecedent basis for this limitation in the claim. 

9. Claim 46 recites the limitation "each of said plurality of clusters" in line 1-2. There is 
insufficient antecedent basis for this limitation in the claim. 

Claim Rejections - 35 USC § 102 

10. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published iinder Article 2 1 (2) of such treaty in the English language. 

11. Claims 1-5, 11, 12, 31-35, 41, and 42 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Ratakonda (USPN 5,956,026, newly cited). 

12. Regarding claims 1 and 3 1 , Ratakonda discloses a method and computer-readable 
medium containing a program for summarizing a content of an input video sequence, said input 




Application/Control Number: 09/556,349 Page 4 

Art Unit: 2623 

video sequence comprising a plurality of frames (column 3, line 21-29), said plurality of frames 
being grouped into a plurality of video segments (column 4, line 48-54), said method and 
program comprising: (a) selecting a frame cluster (represented by keyframe kl) in said input 
video sequence which corresponds to a most static one (minimized action) of said video 
segments (Figure 6; column 6, line 55-60; column 7, Une 25-33); (b) computing a content value 
(cumulative action measure C) in said selected frame cluster (column 6, line 26-44); (c) using 
said computed content value to cluster remaining frames in said input video sequence (column 7, 
line 25-33; column 10, line 35-42). 

13. Regarding claims 2 and 32, Ratakonda discloses that in said step (a), said frame cluster is 
selected using a refined feature space representation (treat each histogram as a feature vector of 
its associated frame) of said input video sequence (column 9, line 44-56). 

14. Regarding claims 3 and 33, Ratakonda discloses that in said step (a), each of said 
plurality of frames is transformed into a histogram vector indicative of a spatial distribution of 
colors in said each of said plurality of frames (column 4, line 48-54). 

1 5. Regarding claims 4 and 34, Ratakonda discloses that in said (a) each of said plurality of 
frames (finest level keyframes) is divided into a plurality of blocks, each of said plurality of 
blocks being represented by a histogram in a color space indicative of a distribution of colors 
within each of said plurality of blocks (column 1 1, line 1-17; column 14, line 15-47). 

16. Regarding claims 5 and 35, Ratakonda discloses that each of said plurality of frames 
(finest level keyframes) is divided into a plurality of blocks and each said histogram vector 
comprises a plurality of histograms in a color space, each of said plurality of histograms 
corresponding to one of said plurality of blocks (column 11, line 1-17; column 14, line 15-47). 
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17. Regarding claims 1 1 and 41, Ratakonda discloses a step (d) including outputting a 
plurality of keyframes, each of said plurality of key frames representative of said clustered 
frames (column 4, line 54-63). 

18. Regarding claims 12 and 42, Ratakonda discloses that said selecting comprises locating a 
cluster (having a histogram vector) closest to an origin (centroid or mean histogram assigned as a 
representative vector) of said refined feature space (column 10, line 5-42). 

Claim Rejections - 35 USC §103 

19. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

20. Claims 6-9 and 36-39 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ratakonda as applied to claims 2, 32, and 33 above, and further in view of Lim (USPN 
6,574,378, newly cited). 

21 . Regarding claims 6 and 36, Ratakonda does not disclose that said refined feature space 
representation is obtained using a singular value decomposition of said input video sequence. 
Lim discloses a method and apparatus for indexing and retrieving images using visual keywords 
wherein during indexing (or retrieval) of a visual document, a spatial aggregation map (SAM) of 
occurrences of visual tokens is created which represents a visual-content signature for the visual 
document (column 9, line 14-28). Subsequently, the SAM is input to a singular- value- 
decomposition (SVD) based coding module to produce a refined feature (reduced 
dimensionality) space representing a coded description of a visual document (column 9, line 29- 
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37). It would have been obvious to one of ordinary skill in the art at the time the invention was 
made to obtain a refined feature space using a singular value decomposition of an input video 
sequence as taught by Lim in order to reduce the dimensionality and possibly the noise in the 
spatial aggregation map to produce a coded description of a visual document (column 9, line 29- 



22. Regarding claims 7 and 37, Ratakonda discloses that minimization (to a refined feature 
space) is performed using frames (keyfi*ames) selected with a fixed interval (equispaced) from 
said input video sequence (column 7, line 42-49). 

23. Regarding claims 8 and 38, Lim discloses that said selected frames (linearized SAM 
vectors) are arranged into a feature frame matrix (X), and wherein said singular value 
decomposition is performed on said feature frame matrix (column 9, line 33-44). 

24. Regarding claims 9 and 39 (see above rejection under 35 U.S.C. 1 12, 2'''^ paragraph), Lim 
discloses that said singular value decomposition produces a matrix (X), each column of said 
matrix representing a frame (SAM vectors of visual documents) in a refined feature (reduced 
dimensionality) space corresponding to a frame in said input video sequence (column 7, line 33- 



25. Regarding claims 17 and 47, Ratakonda discloses a method for summarizing a content of 
an input video sequence (column 3, line 21-29), said method comprising: (a) selecting frames 
(keyframes) from said input video sequence (column 4, line 54-63), said selected frames being 
taken at a fixed (equispaced) interval (column 7, line 42-49); (b) Lim discloses creating a feature 
frame matrix using said selected frames (SAM vectors of visual documents); (c) performing a 
singular value decomposition on said feature frame matrix to obtain a matrix representing said 



33). 



44). 
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video sequence in a refined feature space (see above discussion of claims 1 and 31); (d) 
Ratakonda discloses selecting a cluster (represented by keyframe kl) in said refined feature 
space corresponding to a most static (minimized action) video segment (Figure 6; column 6, line 
55-60; column 7, line 25-33); (e) computing a content value (cumulative action measure C) 
corresponding to said selected cluster (column 6, line 26-44); (f) using said computed content 
value to cluster frames in said input video sequence (column 7, line 25-33; column 10, line 35- 



26. Claims 10 and 40 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ratakonda as applied to claims 1 and 3 1 above, and further in view of Uchihachi et al. (USPN 
6,535,639, newly cited), hereafter Uchichachi. 

27. Regarding claims 10 and 40, Ratakonda does not disclose a step (d) using said clustered 
frames to output a motion video representative of a summary of said input video sequence. 
Uchihachi discloses automatic video summarization using a measure of shot importance and a 
frame-packing method wherein a video may be modified during playback to emphasize the 
measure of shot importance in order to de-emphasize less important shots, or skip shots entirely 
that are less important than a predetermined threshold, resuhing in a summary or "video skim" 
(column 9, line 53-60). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to output a motion video representative of a summary of an input video 
sequence as taught by Uchihachi in order to emphasize the measure of shot importance by de- 
emphasizing or skipping less important shots and also allowing the user to generate a personal 
video summary (column 9, line 53-60). 



42). 
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28. Claims 13-16 and 43 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ratakonda as applied to claims 2 and 32 above, and further in view of Castelli et al. (USPN 
6,122,628, newly cited), hereafter Castelli. 

29. Regarding claims 13 and 43, Ratakonda discloses that said step (c) comprises: (c)(1) 
sorting a plurality of vectors in said refined feature space according to a distance of each of said 
vectors to an origin (representative vector) of said refined feature space representation (column 
9, line 49-column 10, line 20); (c)(2) selecting a vector among said sorted vectors which is 
closest to an origin (representative vector) of said refined feature space representation and 
including said selected vector into a first cluster (column 10, line 35-37); (c)(3) clustering said 
plurality of sorted vectors in said refined feature into a plurality of clusters according to a 
distance between each of said plurality of sorted vectors and vectors in each of said plurality of 
clusters and an amount of information in each of said plurality of clusters (column 10, line 37- 



30. Ratakonda does not explicitly disclose that the vectors are sorted in ascending order. 
Castelli discloses multidimensional data clustering and dimension reduction for indexing and 
searching wherein eigenvalues (characteristic values of a transformation matrix produced by 
singular value decomposition; column 11, line 9-16) are sorted by decreasing magnitude and a 
subset of ordered eigenvalues containing the largest eigenvalues are selected according to a 
selection criterion (column 11, line 25-31). Castelli sorts in decreasing order, but it would have 
been obvious to sort in ascending order, depending on the needs of the system. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to sort a 



42). 
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plurality of vectors in said refined feature space in ascending order as taught by Castelli in order 
to aid in the selection of the vectors (eigenvalues) according to a user-specified selection 
criterion (column 11, line 28-40), 

31. Regarding claim 14, Ratakonda discloses that in said step (c)(3) said plurality of sorted 
vectors are clustered into said plurality of clusters such that said amount of information (action) 
in each of said plurality of clusters does not exceed an amount of information (action) in said 
first cluster (column 6, line 45-60). 

32. Regarding claim 15, Ratakonda discloses that said first cluster is composed of frames 
(keyframes) based on a distance variation between said frames (column 8, hne 52-63) and an 
average distance between frames in said first cluster (same as the distance since the keyframes 
are equispaced; column 7, line 45-49). 

33. Regarding claim 16, Ratakonda discloses that each of said plurality of clusters is 
composed of frames (keyframes) based on a distance variation between said frames (column 8, 
line 52-63) and an average distance between frames in said each of said plurality of clusters 
(same as the distance since the keyframes are equispaced; column 7, line 45-49). 

34. Claims 44-46 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ratakonda 
in view of Lim as applied to claim 38 above (see above rejections under 35 U.S.C. 1 12, 2""** 
paragraph). 

35. Regarding claims 44, 45, and 46, see above rejections of claims 14, 15, and 16, 
respectively. 
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Conclusion 



36. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. USPN 5,619,709 to Caid et al. discloses a system and method of context vector 
generation and retrieval using singular value decomposition. USPN 5,708,767 to Yeo et al. 
discloses a method and apparatus for video browsing based on content and structure. USPN 
6,393,054 to Altunbasak et al. discloses a system and method for automatically detecting shot 
boundary and key frame from a compressed video data. USPN 6,473,095 to Martino et al. 
discloses a histogram method for characterizing video content. USPN 6,496,228 to McGee et al. 
discloses significant scene detection and frame filtering for a visual indexing system using 
dynamic thresholds. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 

If attempts to reach the examiner by telephone are unsuccessfial, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-306-0377. 



rjh 

January 14, 2004 




